Forward- or Reverse-Mode Automatic Differentiation: What's the Difference?

Automatic differentiation (AD) has been a topic of interest for researchers in many disciplines, with increased popularity since its application to machine learning and neural networks. Although appreciate know how apply AD, it remains challenge truly understand the underlying processes. From an algebraic point view, however, AD appears surprisingly natural: originates from laws. In this work we use Algebra Programming techniques reason about different variants, leveraging Haskell illustrate our observations. Our findings stem three fundamental abstractions: (1) notion semimodule, (2) Nagata's construction ‘idealization module’, (3) Kronecker's delta function, that together allow us write single-line abstract definition AD. definition, by instantiating structures various ways, derive have same extensional behaviour, but intensional properties, mainly terms (asymptotic) computational complexity. We show variants equivalent means Kronecker isomorphisms, further elaboration infrastructure which guarantees correctness construction. With framework place, paper seeks make more comprehensible, taking perspective on matter.